In a conventional row-based database, each row (i.e., record) of a database table is stored contiguously in memory. Accordingly, if a new record is added to a table, the values of the new record may be appended to the values of the existing records of the table.
In contrast, a columnar database stores values per table column. FIG. 1 shows table 10, including three columns, and memory locations 20 in which the values of table 10 are stored. Memory locations 20 may represent volatile and/or persisted memory.
The values of column Name are stored in locations beginning with memory location A, the values of column Address are stored in locations beginning with memory location B, and the values of column Telephone are stored in locations beginning with memory location C. More specifically, the values of the first record of table 10 are stored at memory locations A, B and C. Similarly, the values of the fourth record of table 10 are stored at memory locations A+4, B+4 and C+4.
In some cases, a main data structure may be compressed using, for example, a lightweight compression scheme. Updating a compressed structure can be a time consuming process since the structure might need to be unpacked, updated, and re-compressed for each write operation. It may be more efficient to instead collect new records and append them to a delta structure, which stores changes on the table. Once the delta structure reaches a particular size, the data in the delta structure is merged with the actual columnar data of the table (e.g., by adding new values of new records, deleting values of deleted records, and/or updating values of updated records). This merge results in overwriting entire columns, and the delta structure is thereafter empty.
The foregoing process occurs in volatile memory (e.g., Random Access Memory) and in persisted memory (e.g., a hard or solid state disk drive). That is, each of volatile memory and persisted memory include the actual columnar data and a delta structure which is updated on each transaction. During a merge, the actual columnar data of the volatile memory may be merged with the delta structure of the volatile memory and the actual columnar data of the persisted memory is merged with the delta structure of the persisted memory.
As in-memory databases hold the primary persistence in volatile memory, some form of recovery mechanism may be required to avoid a potential data loss in when a failure occurs (e.g., a database crash). Two desirable characteristics of a recovery mechanism are: (1) that it has a minimal impact on the running system and (2) that the system recovers relatively quickly and with little or no data loss after a failure.